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(1) Real Party in Interest 

The real party in interest is International Business Machines Corporation. 

(2) Related Appeals / Interferences 

No other appeals or interferences exist that relate to the present application or 
appeal. 

(3) Status of Claims 

Claims 1 - 3, 5 - 14,16 - 20, 39, 40, and 42 - 46 are pending and remain in the 
application. In the Final Office Action of May 12, 2003 and the Advisory Action of 
June 27, 2003, the claims were indicated to be finally rejected as being unpatentable 
over Jain et al., [hereafter Jain], U.S. Patent No. 5,911,139 in view of Polnerow et al., 
[hereafter Polnerow], U.S. Patent No. 5,813,006. 

(4) Status of Amendments 

No amendments are outstanding. 

(5) Summary of Invention 

The present invention relates to a software program for accessing, transducing, 
enhancing, searching, or otherwise processing various contextual data acquired 
from objects. The present invention automatically associates the acquired contextual 
information w ith available multimedia resources and presenting the results to the 
user. 

The term " contextual " has been clearly defined as follows: "As used herein, 
"contextual" means or implies the surrounding circumstances in which an 
obiect exists or an event occurs . For example, the contextual content of a 
photograph can be all the information surrounding the situation in which the 
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photograph is taken, including special and physical parameters such as time, 
location, elevation, etc., as well as information gathered by various sensors such as 
temperature, pressure, humidity, light level, sound, and acceleration sensors, and 
user interface elements such as buttons, switches, etc. The contextual data helps to 
understand the context of the acquired data." Reference is made to page 5, lines 5 - 
12 of the specification. 

The present system includes a contextual multimedia association module which 
is installed on a contextual input device. In use, the user captures input data about 
an object or item of interest by means of the contextual input device, and transmits 
the captured data to the personal assistant device. The personal assistant device 
automatically digitizes and processes the input data, and further automatically 
formulates a query, creates or updates a digital user profile, and transmits the query 
to the contextual multimedia association module. The contextual multimedia 
association applies the query to numerous data stores, optimizes the search results, 
and then presents the optimized search results to the user. 

The present system provides various features, including but not limited to the 
following: 

Image magnification with anti-jitter or jitter reduction feature. 

Image search capability, whereby the system automatically formulates a query, 
searches for, and matches the image content of an object to images in various data 
stores. This image search capability allows the system and method to be used in 
various commercial applications such as: 
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information access based on data acquired from real objects. 

* The association of on-line "virtual" information with physical "real" objects. 

* E-commerce : The system enables on-line shopping in-situ based on imaged 

items. 

' Remote medical and treatment : For example, a dermatologist can image a 
skin rash, and the system compares a skin legion to known diseases and 
acquires additional information from various online medical databases, 
• Security, personal information , electronic validation of information to create a 
"digital witness", digital notary public, digital signature capture, insurance: For 
example, facial recognition of a customer can be taken at the time of sale. As 
another example, a signature can be captured or insurance photos taken and 
authenticated by time and location. As another example, the system can 
validate images and their authenticity. 

• Advanced information access in museums : For example, images of art works 
can be acquired by a curator, and the system will recognize these works and 
present relevant details. 

• Automotive diagnostic : For example, a mechanic can image a car vehicle 
identification number (VIN) and take a picture of the part of interest. The 
system will match the images and display relevant pages from shop manuals. 
• Triage : For example, damages and / or injuries can be taken at an accident 
site, and the system can perform an automatic search and sorting (e.g. triage) 
of patients, suggest courses of action, and request authorization for 
settlement, repair, indemnification, medical care, and so forth. 

Expanded optical character recognition (EOCR) of text acquired from real objects, 
such as books, road signs, shopping, and so forth, with automatic Internet search 
capability. 
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Education, training and translation; for example, students can input a query, or the 
system can automatically formulate a query from an imaged object of interest, and 
the system will OCR the relevant text, perform translation if needed, and undertake 
an in-depth search of data stores around the world, even if the information is stored 
in different languages. The search results are translated to the language of interest 
and presented to the student. This feature can also be used for in-store shopping. 
For example, when shopping for books, book titles could be scanned, and related 
books will be displayed. 

Audio to text search capability, whereby sounds, such as animal sounds, are 
captured and compared to sounds in various data stores to identify the animal specie 
or other requested information about the source of the audio signal. 

Spatial and temporal recordation of events, wherein for example, a user can 
combine visual information with virtual or invisible information such as GPS. In one 
application, the system allows the user to spatially record sound, and whenever the 
user reaches a destination location the system reminds the user by playing the 
recorded message. 

Real-time solution of mathematical or geometric problems, real time problem 
solving. For example, this feature can be used to balance check books, to convert 
currencies, and in a variety of similar other applications. In another specific example, 
the system can assist the user in solving geometric problems using the imaged 
information and present solutions to the user. Practical applications involve home 
repair and replacing parts when dimensions are not known. 

Mass spectrometry: For example, a user can acquire chemical and other data 
about a material, and the system will automatically formulate and apply a search for 
identifying the material. 
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(6) Issue Presented for Review 

The issue for review is whether claims 1 - 3, 5 - 14,16 - 20, 39, 40, and 42 - 46 
are obvious in light of Jain in view of Polnerow. 

(7) Grouping of Claims 

Claims 1 - 3, and 5 - 12 are grouped together and stand and fall together. 

Claims 13, 14, and 16-20 are grouped together and stand and fall together. 
Claims 39. 40, and 42 - 46 are grouped together and stand and fall together. 

(8) Arguments 
A - introduction 

The issue under review is whether claims 1 - 3, 5 - 14,16 - 20, 39, 40, and 42 - 46 
are obvious in light of Jain in view of Polnerow, 

Applicant respectfully traverses the obviousness rejection of these claims, and 
submits that the claims on file are not obvious in view of Jain in light of Polnerow, and 
are patentable thereover. In support of this position, Applicant submits the following 
arguments: 

B ■ Legal Standards for Obviousness 

The following legal authorities set the general standards in support of Applicants 
position of non obviousness, with emphasis added for added clarity: 

• MPEP §2143.03, "All Claim Limitations Must Be Taught or Suggested: To 
establish prima facie obviousness of a claimed invention, all the claim 
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limitations must be taught or suggested bv the prior art . In re Royka, 490 
F.2d 981, 180 USPQ 580 (CCPA 1974). "All words in a claim must be 
considered in judging the patentability of that claim against the prior art." In re 
Wilson, 424 F.2d 1382, 1385, 165 USPQ 494, 496 (CCPA 1970). If an 
independent claim is nonobvious under 35 U.S.C. 103, then any claim depending 
therefrom is nonobvious. In re Fine, 837 F.2d 1071, 5 USPQ2d 1596 (Fed. Cir. 
1988)." 

• MPEP §2143.01 , "The Prior Art Must Suggest The Desirability Of The Claimed 
Invention: There are three possible sources for a motivation to combine 
references: the nature of the problem to be solved, the teachings of the prior art, 
and the knowledge of persons of ordinary skill in the art." In re Rouffet, 149 F.3d 
1350, 1357, 47 USPQ2d 1453, 1457-58 (Fed. Cir. 1998) (The combination of 
the references taught every element of the claimed invention, however 
without a motivation to combine, a rejection based on a prima facie case of 
obvious was held improper ). The level of skill in the art cannot be relied upon 
to provide the suggestion to combine references. Al-Site Corp. v. VSI Int'l Inc., 
174 F.3d 1308, 50 USPQ2d 1161 (Fed. Cir. 1999). 

• " Obviousness cannot be established by combining the teachings of the prior art 
to produce the claimed invention, absent some teaching or suggestion 
supporting the combination." In re Fine, 837 F.2d at 1075, 5 USPQ2d at 1598 
(citing ACS Hosp. Sys. v. Montefiore Hosp., 732 F.2d 1572, 1577, 221 USPQ 
929, 933 (Fed. Cir. 1984)). What a reference teaches and whether it teaches 
toward or awav from the claimed invention are questions of fact. See Raytheon 
Co. V. Roper Corp., 724 F.2d 951, 960-61, 220 USPQ 592, 599-600 (Fed. Cir. 
1983), cert, denied, 469 U.S. 835, 83 L. Ed. 2d 69, 105 S. Ct. 127 (1984). " 

• "When a rejection depends on a combination of prior art references, there must 
be some teaching, suggestion, or motivation to combine the references. See 
In re Geiger, 815 F.2d 686, 688, 2 USPQ2d 1276, 1278 (Fed. Cir. 1987)." 
Obviousness can only be established bv combining or modifying the teachings of 
the prior art to produce the claimed invention where there is some teaching, 
suggestion, or motivation to do so found either explicitly or implicitly in the 
references themselves or in the knowledge generally available to one of ordinary 
skill in the art. See MPEP 2143.01; In re Kotzab, 217 F.3d 1365, 1370, 55 
USPQ2d 1313, 1317 (Fed. Cir. 2000); In re Fine, 837 F.2d 1071, 5 USPQ2d 1596 
(Fed. Cir. 1988); and In re Jones, 958 F.2d 347, 21 USPQ2d 1941 (Fed. Cir. 
1992). 

• "With respect to core factual findings in a determination of patentability, however, 
the Board cannot simpiv reach conclusions based on its own understanding 
or experience ~ or on its assessment of what would be basic knowledge or 
common sense. Rather, the Board must point to some concrete evidence in 
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the record in support of these findings." See In re Zurl<o, 258 F.3d 1379 (Fed. 
CIr. 2001). 

• "We have noted that evidence of a suggestion, teaching, or motivation to 
combine may flow from the prior art references themselves, the knowledge of 
one of ordinary skill in the art, or, in some cases, from the nature of the problem to 
be solved, see Pro-Mold & Tool Co. v. Great Lakes Plastics, Inc., 75 F.Sd 1568, 
1573, 37 USPQ2d 1626, 1630 (Fed. Cir. 1996), Para-Ordinance Mfg. v. SGS 
Imports Intern., Inc., 73 F.3d 1085, 1088. 37 USPQ2d 1237, 1240 (Fed. Cir. 
1995), although "the suggestion more often comes from the teachings of the 
pertinent references," Rouffet, 149 F.3d at 1355, 47 USPQ2d at 1456. The range 
of sources available, however, does not diminish the requirement for actual 
evidence. That is, the showing must be clear and particular . See, e.g., C.R. 
Bard, 157 F.3d at 1352, 48 USPQ2d at 1232. Broad conciusorv statements 
regarding the teaching of multiple references, standing alone, are not 
"evidence." E.g.. McElmurry v. Arkansas Power & Light Co., 995 F.2d 1576, 
1578, 27 USPQ2d 1129, 1131 (Fed. Cir. 1993) ("Mere denials and conclusory 
statements, however, are not sufficient to establish a genuine issue of material 
fact."); In re Sichert, 566 F.2d 1154, 1164, 196 USPQ 209, 217 (CCPA 1977)." 
See In re Dembiczak, 175 F. 3d 994 (Fed. Cir. 1999). 

• "To prevent the use of hindsight based on the invention to defeat patentability of 
the invention, this court reguires the examiner to show a motivation to 
combine the references that create the case of obviousness. In other words, the 
examiner must show reasons that the skilled artisan, confronted with the same 
problems as the inventor and with no knowledge of the claimed invention, would 
select the elements from the cited prior art references for combination in the 
manner claimed " See In re Rouffet, 149, F.3d 1350 (Fed. Cir. 1998). 

• The mere fact that references can be combined or modified does not render the 
resultant combination obvious unless the prior art also suggests the 
desirability of the combination . In re Mills, 916 F.2d 680, 16 USPQ2d 1430 
(Fed. Cir. 1990). Although a prior art device "may be capable of being modified to 
run the way the apparatus is claimed, there must be a suggestion or 
motivation in the reference to do so." 916 F.2d at 682, 16 USPQ2d at 1432.). 
See also In re Fritch, 972 F.2d 1260, 23 USPQ2d 1780 (Fed. Cir. 1992) (flexible 
landscape edging device which is conformable to a ground surface of varying 
slope not suggested by combination of prior art references). 

• If the proposed modification would render the prior art invention being 
modified unsatisfactorv for its intended purpose, then there is no suggestion or 
motivation t o make the proposed modification. In re Gordon, 733 F.2d 900, 221 
USPQ 1125 (Fed. Cir. 1984). 
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Jain generally describes a system and method for content-based search and 
retrieval of visual objects. A base visual information retrieval (VIR) engine utilizes a 
set of universal primitives to operate on the visual objects. An extensible VIR engine 
allows custom, modular primitives to be defined and registered. A custom primitive 
addresses domain specific problems can utilize image understanding techniques. 
Object attributes can be extracted over the entire image or over only a portion of the 
object. A schema is defined as a specific collection of primitives . A specific 
schema implies a specific set of visual features to be processed and a corresponding 
feature vector to be used for content-based similaritv scoring. A primitive registration 
interface registers custom primitives and facilitates storing of an analysis function and 
a comparison function to a schema table. A heterogeneous comparison allows 
objects analyzed by different schemas to be compared if at least one primitive is in 
common between the schemas. A threshold-based comparison is utilized to improve 
performance of the VIR engine. A distance between two feature vectors is computed 
in any of the comparison processes so as to generate a similarity score. Refer to the 
Abstract. 

"The VIR Engine 120 comprises two main modules: an "Image Analysis" module 
122 and an "Image Comparison" module 124. The image analysis module 122 
receives inputs from either module 108 or 1 10 to generate a query target or from the 
insertion module 1 12 for adding a new image into the database 132. The output of 
the image analysis module 122 is a feature vector (FV) that describes the visual 
object passed to it by one of modules 108, 1 10 or 1 12. The FV is passed on to the 
database engine 130. In addition, if module 112 was used to insert the image into the 
database, both the FV for the image and the image itself are stored in the database 
132 (as seen in FIG. 5B), The image analysis module 122 will be described in greater 
detail hereinbelow. 
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The image comparison module 124 receives a query target FV and a FV for the 
image being tested or compared from the database engine 130. The output of the 
image comparison module 124 is a similarity score that is sent to a "Ranked List 
Management" module 134. A plurality of images from the database 132 are 
compared one at a time to the query image by the image comparison module 124. 
The resultant similarity scores are accumulated by the module 134 so as to provide a 
rank in an order of their similarity to the query image. The ranked results of the list 
management module 134 are provided to a "Thumbnail Results Browser" 136 for 
display to the user 102 through the computer I/O 104. An exemplary screen display 
of ranked results is shown in FIG. 4." Reference is made to FIG. 1 A, and further to 
column 9, lines 41 - 67. 

" What is needed is a new multimedia information system technology model such 
as a visual information management system (VIMSYS) model. Unlike traditional 
database systems, this model recognizes that most users prefer to search image 
and video information bv what the image or video actually contains, rather than 
bv keywords or descriptions associated with the visual information . The only 
proper method by which the user can get access to the content of the image is by 
using image-analysis technology to extract the content from an image or video . 

Once extracted, jthg„icfl!ntgnt rsBrssgnfe mft§t 9f What th? M^?r n^?ti.§„in vrtigr t9 
ftraamzgi §g,arfih...an,d l9,gatg„nfigg§8arY.,Y,i§Mal init9.rniatifto. 

This breakthrough concept of content extraction alleviates several technological 
problems. The foremost benefit is that it gives a user the power to retrieve visual 
information by asking a query like "Give me all pictures that look like this." The 
system satisfies the query by comparing the content of the guery picture w ith that 
of all target pictures In the database. This is called Query By Pictorial Example 
(QBPE), and Is a simple form of content-based retrieval , a new paradigm in 
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database management systems." Reference is mad to column 2, lines 53 - 65, with 
emphasis added. 

D- Application of the Obviousness Standard to the Present Invention 

D.1 ■ Elements not disclosed bv Jain 

Applicant respectfully submits that the following elements and combination of 
elements and resulting features recited in the representative independent claim 1 and 
the claims dependent thereon, are not disclosed in Jain (with emphasis added): 

"1 . A system for automaticallv associating contextual input data w ith 
available multimedia resources, comprising: 

a contextual input device for capturing the contextual input data : 

an assistant device for processing the contextual input data captured 
by the contextual input device, and for formulating a querv based on 
processed contextual input data and a user profile : and 

a contextual multimedia association module for associating the 
processed contextual Input data w ith the multimedia resources and for 
generating association matches." Emphasis added. 

D.2. Summarv of Arguments 

Applicant respectfully submits a summary of the arguments to be developed in 
more detail in subsequent sections: 
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D.2.a: Jain does not capture contextual input data. 

D.2.b: Jain does not formulate a query based on the processed contextual input 
data and the user profile. 

D.2,a: Jain does not capture contextual input data 

As it is clear from the Jain excerpts above, Jain's method is strictly content- 
based , in that it is limiting to discovering and cataloguing the content of the input 
image . Jain does not capture contextual input data , as understood and defined in 
the present application. To this end, and due to the importance of the concept of 
contextual data versus content, Applicant has once again reproduced one of the 
excerpts of Jain: "... using image-analysis technology to extract the content from 
an image or video. Once extracted, .ttlg.fignt^nt »:gPIgSfinfe,m,9,§t 9f Wh^t thff 

In essence, Jain has clearly stated that the content represents most of what 
the user needs , thus expressly distinguishing over the contextual input data. 

The term "contextual" is a kev concept to the present Invention, and is recited 
eight times in the instant claim 1. Applicant respectfully requests that this term be 
given its proper weight and not be summarily dismissed. The term "contextual" is 
clearly defined in the present application as follows: 

"As used herein, "contextuai" means or implies the surrounding 
circumstances in which an object exists or an event occurs . For example. 
the contextual content of a photograph can be all the information surrounding 
the situation in which the photograph is taken, including special and physical 
parameters such as time, location, elevation, etc. . as well as information 
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gathered by various sensors such as temperature, pressure, humidity, light 
level, sound, and acceleration sensors, and user interface elements such as 
buttons, switches, etc. The contextual data helps to understand the context of 
the acquired data," Reference is made to page 7, lines 5 - 12 of the present 
application, with emphasis added for clarity. 

" Exemplary , non-exclusive contextual data include environmental and 
qeospatial coordinates, time, temperature, location, speed, motion, 
acceleration, and other parameters ." Reference is made to page 7, lines 1 - 3 
of the present application, with emphasis added for clarity. 

The concept of capturing contextual input contextual input data, and then 
processing the captured contextual input data is neither taught nor suggested in Jain. 
Jain's system is content-based, as opposed to context-based teaching of the 
present invention , in that Jain seems to disregard the surrounding 
circumstances, such as the environmental and geospatial coordinates, time, 
temperature, location, speed, motion, or acceleration in which the content exists or 
the event occurs . 

The Advisory Action states that: "In response to the independent claims, Jain 
clearly teaches the applicant's argument of "contextual" information (see col. 7, lines 
27-53; Fig. 3; and col. 11, lines 10-17). Although Jain teaches of content-based, such 
limitation in the reference does not exclude that his system cannot be contextual or 
deal with contextual data, since the definition of content and context are not 
opposite." 

It is simply not clear to Applicant where do the foregoing cited sections of Jain, 
i.e., col.7, lines 27-53; Fig. 3; and col.11, lines 10-17, describe contextual information. 
Once again, Applicant wishes to reiterate that the term "contextual" has been 
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clearly defined in the instant application, and must NOT be redefined by the 
Examiner. 

The Examiner refers to " applicant's argument of "contextual" information ". 
However, Applicant's consistent position and definition of the term "contextual" has 
been the surrounding circumstances in which an object exists or an event 
occurs . For example, the contextual content of a photograph can be all the 
information surrounding the situation in which the photograph is taken, including 
special and physical parameters such as time, location, elevation, etc. , as well as 
information gathered by various sensors such as temperature, pressure, humidity, 
light level, sound, and acceleration sensors, and user interface elements such as 
buttons, switches, etc. 

Applicant respectfully submits that the Examiner has clearly failed to prove that 
Jain describes contextual information. In fact, Applicant's second statement above, 
seems to imply that Jain has taught the content-based concept, and since content- 
based limitation " does not exclude that his system cannot be contextual or deal with 
contextual data, since the definition of content and context are not opposite ," then 
Jain describes the contextual information. 

In response. Applicant rejects the Examiner's argument and submits that the 
Examiner has not met his burden of proving prime facie obviousness, in that: 

1 . The Examiner has neither expressly nor by implication shown that Jain teaches 
the use of contextual information as defined by the present application. 

2. The reasoning that a concept is described because it is not excluded is neither 
tenable nor justifiable, and does not meet the scrutiny of the well settled legal 
authorities above. 

3. Jain expressly states that: the content represents most of what the user needs, 
thus expressly distinguishing over the contextual input data. 
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D, 2,b. Jain does not teach automaticailv 



formulating the query based on the 



contextual input data and the user profile 



M) Applicant agrees with the Examiner that Jain does not teach automatically 
formulating the query based on the user profile. 

(2) Furthermore, Applicant submits that Jain does not teach automatically 
formulating the query based on the contextual input data , since, as stated earlier, 
Jain seems to ignore the contextual data, and rather focuses on the content data. 

In order to compensate for the first missing element, the office action resorts to 
Polnerow, as discussed below. 

D. 3. The combination of Jain and Polnerow is improper 

In support of the combination of Jain and Polnerow, the office action adds that 
Polnerow teaches searching based on user profile (see col. 1, lines 55-65 and 
column 10, lines 53-54), reasoning that: 

"It would have been obvious to a person of ordinary skill in the art at the 
time the invention was made to employ the teachings of Polnerow within the 
system of Jain by automatically searching database for information based on 
user profile within the contextual input data and multimedia data system 
associating system and method because. Jain teaches of user specific 
property or attribute of the image f or performing queries (see col.4, lines 21- 
40) and such attributes could be users preference according to the users 
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profile. Since user supplies the property or attributes, theses attributes or 
properties could be derived according to users preference such as users 
profile ." 

Applicant respectfully traverses this rejection ground and submits the following 
arguments: 

(1) Jain limits the "specific property or attribute of the image" to what is referred to as 
"primitives" of the content. See for example, column 6, lines 20-21 in Jain. There is, 
however, no justification or teaching in Jain to expand the "specific property or 
attribute of the image" to cover "the user's profile," particularly that Jain does not 
consider the user's profile to be an attribute of the image. This latter point is 
discussed in more detail below. 

(2) In addition, Polnerow does not disclose formulating the query based on the user's 
profile " for automatically associating contextual input data w ith available 
multimedia resources." 

(3) Furthermore, neither Polnerow nor Jain teaches or suggests "formulating a query 
based on processed contextual input data ," as presented earlier, and as clearly 
defined in the present application. 

(4) The Examiner bases the ground for combining Jain on Polnerow on the user's 
profile, but does not provide any ground for combining these two references based 
on the processed contextual input data . Applicant submits that these two 
references cannot be justifiably combined based on the processed contextual input 
data , since this feature is neither suggested nor taught by any of these two 
references. 
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(5) In the Advisory Action , the Examiner states the following ground in support of 
combining the prior art references: 

"2. In response to applicant's argument that there is no suggestion to 
combine the references, ... In this case, all the references teach of means 
for searching for information by means of querying ." 

3. In response to applicant's arguments against the references individually, 
one cannot show nonobviousness by attacking references individually where 
the reiections are based on combinations of references . . (Emphasis 
added). 

In response to Paragraph 2 above, it is not clear to the Applicant how the fact that 
the references teach searching by query justifies the combination of these two 
references! Applicant is not familiar with the legal authority on which the Examiner 
bases this rejection ground. Does the Examiner mean to say that just because the 
two cited references teach searching by query they could be combined? 

In response to Paragraph 3 above, Applicant's arguments clearly address the 
combination of the two references. Prior to discussing the combination as stated by 
the Examiner, Applicant would need to concede that such a combination is justifiable. 
Applicant does not make such an admission, and clearly states that the combination 
of the two cited references is not justified under the prevailing legal authorities above. 

It is therefore not clear to Applicant how Applicant should "attack" or distinguish 
over a combination that is not justified ab initiol 

As a result, claim 1 and the claims dependent thereon are allowable, and such 
allowance is respectfully requested. Similarly, independent claims 13 and 39 and the 
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claims dependent thereon are allowable for reciting comparable elements to those of 
claim 1. 

E. Dependent Claims 

E. 1. Claims 7, 17, and 43 

Claims 7, 17, and 43 were rejected under 35 U.S.C. 103(a) as being unpatentable 
over Jain in view of Bull. 

Applicant respectfully traverses this rejection and submits that the rejected claims 
are not obvious in view of Jain and Bull, and are thus patentable thereover. In 
support of this position, Applicant submits Bull was referred to as teaching the 
development of a user profile. 

Applicant respectfully submits that the same arguments presented earlier in favor 
of allowance of claim 1 over Jain and Polnerow are equally applicable to the 
combination of Jain and Bull. Further, Bull stands for the generally known concept of 
developing a user profile, which, by itself, does not constitute the novel and 
patentable aspect of the present invention. 

The combination of Jain and Bull does not disclose the following elements of the 
representative claim 7: 

• "formulating a query based on processed contextual input data and a user 
profile"; and 

• "the assistant device develops the user profile based on association matches 
that were previously presented to the user". 
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Claim 7 and the claims dependent thereon are allowable, and such allowance is 
respectfully requested. Similarly, independent claims 17 and 43 and the claims 
dependent thereon are allowable for reciting comparable elements to those of claim 
7. 

E. 2. Claims 8. 18. and 44 

The office action states that with regard to claims 8, 18, and 44, "Jain does not 
teach wherein the assistant device updates the user digital profile based on 
recent association matches . Bull teaches of updating the user digital profile based 
on recent association matches (see col. 8, line 65 to col.9, line 2 and col. 12, lines 2- 
4). It would have been obvious to a person of ordinary skill in the art at the time the 
invention was made to employ the teachings of Bull within the system of Jain by 
updating user profile within the contextual input data and multimedia data system 
associating system and method because, this would keep track of the potential future 
user as well as keep track of the user's preferences for future search "primitives" to 
speed up the processing time." Emphasis added. 

Applicant respectfully submits that the following elements and combination of 
elements and resulting features recited in the representative dependent claim 8 and 
the claims dependent thereon, are not disclosed in Jain (with emphasis added): 

"8. The system according to claim 7, wherein the assistant device updates 
the user digital profile based on recent association matches ." Emphasis 
added 

Applicant incorporates herein the presentation made earlier in favor of the 
allowance of claim 7, and further respectfully submits that claim 8 depends on claims 



AM9-99-0227 



19 



1 and 7 as now amended, to include the following elements that are missing from 
Jain and Bull: 

• "formulating a query based on processed contextual input data and a user 
profile"; 

• "the assistant device develops the user profile based on association matches 
that were previously presented to the user"; and 

• "the assistant device updates the user digital profile based on recent 
association matches". 

Claim 8 and the claims dependent thereon are allowable, and such allowance is 
respectfully requested. Similarly, independent claims 18 and 44 and the claims 
dependent thereon are allowable for reciting comparable elements to those of claim 
8. 

E. 3. Remaining dependent claims 

The remaining dependent claims are allowable for depending on the allowable 
claims 1,13, and 39, as presented earlier. 
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CONCLUSION 



All the claims presently on file in the present application are in condition for 
immediate allowance, and such action is respectfully requested. 

Respectfully submitted, 



Date: August 12. 2003 
Samuel A. Kassatly Law Office 
6819 Trinidad Drive 
San Jose, OA 95120 
Tel. (408) 323-51 1 1 
Fax; (408) 323-5112 




Samuel A. Kassatly 
Attorney for Applicant 
Reg. No. 32,247 
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APPENDIX A 



CLAIMS ON APPEAL 

1. A system for automatically associating contextual input data with available 
multimedia resources, comprising: 

a contextual input device for capturing the contextual input data; 

an assistant device for processing the contextual input data captured by the 
contextual input device, and for formulating a query based on processed contextual 
input data and a user profile; and 

a contextual multimedia association module for associating the processed 
contextual input data with the multimedia resources and for generating association 
matches. 

2. The system according to claim 8, wherein the assistant device automatically 
formulates the query. 

3. The system according to claim 1 , wherein the assistant device automatically 
formulates the query based on a contextual input from the user. 

5. The system according to claim 3, wherein the contextual input device digitizes 
the contextual input data. 

6. The system according to claim 5, wherein the assistant device presents the 
association matches to a user. 

7. The system according to claim 3, wherein the assistant device develops the 
user profile based on association matches that were previously presented to the 
user. 
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8. The system according to claim 7, wherein the assistant device updates the user 
digital profile based on recent association matches. 

9. The system according to claim 8, wherein the contextual multimedia 
association applies the query to a data store on a network, 

10. The system according to claim 9, wherein the network includes the World 
Wide Web. 

1 1 . The system according to claim 7, wherein the contextual input data are based 
on image signals; and 

wherein the assistant device enhances the quality of the image signals. 

12. The system according to claim 7, wherein the contextual input data are based 
on audio signals; and 

wherein the assistant device enhances the quality of the audio signals. 

13. A method for automatically associating contextual input data with available 
multimedia resources, comprising: 

capturing the contextual input data; 
processing the contextual input data; 

formulating a query based on processed contextual input data and a user profile; 
and 

associating the processed contextual input data with the multimedia resources 
and generating association matches. 

14. The method according to claim 18, wherein formulating the query comprises 
automatically formulating the query based on a contextual input from the user. 
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16. The method according to claim 13, further comprising presenting the 
association matches to a user. 

17. The method according to claim 16, further comprising developing a digital 
profile for a user based on association matches which were previously presented to 
the user. 

18. The method according to claim 17, wherein developing the digital profile 
comprises updating the user digital profile based on recent association matches. 

19. The method according to claim 18, wherein associating the processed 
contextual input data comprises applying the query to a data store on a network. 

20. The method according to claim 18, wherein the contextual input data are 
based on any one or more of image signals or audio signals; and 

wherein processing the contextual input data comprises enhancing the quality of 
the any one or more of image signals or audio signals. 

39. A system for automatically associating contextual input data with available 
multimedia resources, comprising: 

means for capturing the contextual input data; 

means for processing the contextual input data and formulating a query based on 
processed contextual input data and a user profile; and 

means for associating the processed contextual input data with the multimedia 
resources and generating association matches. 
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40. The system according to claim 42, wherein the means for processing and 
formulating the query comprises means for automatically formulating the query based 
on a contextual input from a user. 

42. The system according to claim 39, further comprising means for presenting 
the association matches to a user. 

43. The system according to claim 42, further comprising means for developing a 
digital profile for a user based on association matches which were previously 
presented to the user. 

44. The system according to claim 43, wherein the means for developing the 
digital profile comprising means for updating the user digital profile based on recent 
association matches. 

45. The system according to claim 42, wherein the means for associating the 
processed contextual input data comprises means for applying the query to a data 
store on a network. 

46. The system according to claim 42, wherein the contextual input data are 
based on any one or more of image signals or audio signals; and 

wherein the means for processing the contextual input data comprises means for 
enhancing the quality of the any one or more of image signals or audio signals. 
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